4.4. Evaluations
In one glance
- You will: Calibrate a deterministic grader and record its evidence before running live evaluations.
- You need: Workshop step 6 completed; install:eval for the optional MLflow recording.
- Time: about 25 minutes, hands-on.
What should you measure first?
Measure whether the answer contains the correct incident IDs, including no invented IDs.
mise run lab -- check 6
The fixture contains correct, incomplete, and invented answers. Your grader must agree with all labels. This is grader calibration: testing the measurement itself before using it to compare models. It does not prove the agent answers correctly.
How do you record the result in MLflow?
Add MLflow after you have a dataset and a result worth recording.
cd agents/python
mise run install:eval
cd ../..
mise run lab -- record 6
Expected: an offline calibration run ID and a local learning/evaluations.db. The run records label agreement, the Python source, and the cases. It starts no model, container, or server. Open the local result when you want a UI:
cd agents/python
uv run --group eval mlflow ui --backend-store-uri sqlite:///../../learning/evaluations.db --host 127.0.0.1 --port 5000
Stop it with Ctrl-C. MLflow answers which change produced which result. Keep prompts in Git initially; the optional registry is covered later.
How do offline tests and model evaluations differ?
Offline tests verify deterministic behavior; model evaluations measure sampled behavior from a particular configuration.
| Evidence | What it can establish |
|---|---|
| Python tests | Tool behavior, validation, state isolation, approval enforcement |
| Recorded answer calibration | Whether a grader catches known correct and incorrect answers |
| Live evaluation | Observed outcomes for a named model, prompt, dataset, and run |
| Repeated held-out evaluation | Variance and regression evidence beyond the development examples |
| Platform smoke | Whether the deployed transport and configuration work together |
Save development and held-out cases separately. Define acceptance before changing prompts. Require every critical approval and evidence case to pass, then inspect aggregate task success, latency, calls, and token usage.
How do you evaluate the completed reference?
Use the same reference dataset before and after the platform handoff.
Live evaluation consumes model quota
The commands below call the configured model. Gemini can be billed and free-tier quota may interrupt a run. Run a small bounded campaign deliberately; do not interpret a quota failure as an incorrect answer.
cd agents/python
mise run eval:validate # offline fixture validation
mise run eval # live reference cases
The live gate now requires at least 80% aggregate success plus every named critical case. This is an acceptance target, not a claimed measured result. The older 33% collapse-detection floor is not a course completion standard. Baseline artifacts retain their original observed results.
Every case also records what the answer cost to produce. ADK reports four efficiency metrics automatically, next to the trajectory score, in each result file under agents/python/src/agent/.adk/eval_history/:
| Metric | What it counts for one case |
|---|---|
tool_call_count_v1 |
Tool calls the agent made |
inference_call_count_v1 |
Model calls across the whole turn, sub-agents included |
token_usage_v1 |
Tokens, split into input, output, cached, and reasoning |
invocation_duration_v1 |
Wall-clock seconds for the turn |
They are informational: they never pass or fail a case, and ADK rejects a threshold on them. Compare them across runs with the same model to catch a correct-but-expensive regression, an answer that still passes while its calls or tokens quietly double.
For repeated workflow runs, MLflow tracing, groundedness, cost comparisons, model judges, and prompt registries, consult 4.7. Evaluation Reference. Those mechanisms remain available without becoming first-exercise prerequisites.
What proves this page worked?
Run from the repository root; return there if you entered agents/python for the optional reference evaluation.
mise run lab -- check 6
You are done when:
- Your grader rejects known missing and invented evidence.
- You can locate the labeled cases and distinguish calibration from agent-quality evidence.
- If you recorded a run, it identifies the exact source and cases used.
- You defined critical cases and acceptance thresholds before a live model comparison.
Continue to 4.8. Developer Handoff when you can explain the evidence you have and the model behavior you still need to measure.